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Abstract  Intelligent  virtual  agents  are  typieally  embedded  in  a  soeial  environment 
and  must  reason  about  soeial  cause  and  effect.  Social  causal  reasoning  is  qualitatively 
different  from  physical  causal  reasoning  that  underlies  most  current  intelligent  sys¬ 
tems.  Besides  physical  causality,  the  assessments  of  social  cause  emphasize  epistemic 
variables  including  intentions,  foreknowledge  and  perceived  coercion.  Modeling  the 
process  and  inferences  of  social  causality  can  enrich  believability  and  cognitive  capabili¬ 
ties  of  social  intelligent  agents.  In  this  report,  we  present  a  general  computational  model 
of  social  causality  and  responsibility,  and  empirical  results  of  a  preliminary  evaluation  of 
the  model  in  comparison  with  several  other  approaches. 
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1. 


Introduction 


Research  in  intelligent  virtual  agents  has  emphasized  human-like  qualities  in  the  physical 
manifestation  of  agents,  but  such  realism  is  typically  skin-deep.  Although  agents  can  in¬ 
teract  in  naturalistic  ways  with  human  users  and  can  successfully  mimic  speech,  body 
language  and  even,  the  core  reasoning  techniques  that  drive  such  behaviors  have  not  fun¬ 
damentally  changed.  Most  intelligent  systems  incorporate  planning  and  reasoning  tech¬ 
niques  designed  to  reason  about  physical  causality.  Unfortunately,  physical  causes  and 
effects  are  simply  inadequate  for  exploiting  and  explaining  social  phenomena.  In  con¬ 
trast,  social  causality,  both  in  theory  and  as  practiced  in  everyday  folk  judgments  and  in 
the  legal  system,  emphasizes  multiple  causal  dimensions,  incorporates  epistemic  vari¬ 
ables,  and  distinguishes  between  cause,  responsibility  and  blame. 

Recent  approaches  to  social  causality  have  addressed  these  differences  by  extending 
causal  models  [Halpern  &  Pearl,  2001;  Chockler  &  Halpem,  2004],  although  it  is  unclear 
whether  a  full  accounting  of  social  causality  will  (or  even  should)  result  from  such  exten¬ 
sions.  In  contrast,  we  start  with  social  causality  theory  and  consider  how  this  could  be 
formalized  in  a  computational  model.  This  allows  intelligent  entities  to  reason  about  as¬ 
pects  of  social  causality  not  addressed  by  extended  causal  models  and  provides  a  com¬ 
plementary  perspective  to  the  enterprise  of  causal  reasoning  about  social  events. 

Psychological  and  philosophical  theories  identify  key  variables  that  mediate  determina¬ 
tions  of  social  causality.  In  these  theories,  social  causality  involves  not  only  physical  cau¬ 
sality,  with  an  emphasis  on  human  agency,  but  also  people’s  freedom  of  choice  (e.g., 
coercion  [Shaver,  1985]  and  controllability  [Weiner,  1995]),  intentions  and  foreknowl¬ 
edge  [Shaver,  1985;  Zimmerman,  1988].  Using  these  variables,  social  causality  makes 
several  distinctions  not  present  in  the  determinations  of  physical  cause.  For  example,  an 
actor  may  physically  cause  an  event,  but  be  absolved  of  responsibility  and  blame.  Or  a 
person  may  be  held  responsible  and  blameworthy  for  what  she  did  not  physically  cause. 

Our  goal  is  to  model  the  underlying  process  and  inferences  of  social  causality  to  enrich 
the  cognitive  and  social  functionality  of  intelligent  agents.  Such  a  model  can  help  an 
agent  to  explain  the  observed  social  behavior  of  others,  which  is  crucial  for  successful 
interactions  among  social  entities.  It  can  enrich  the  design  components  of  human-like 
agents,  guide  strategies  of  natural  language  conversation  and  model  social  emotions 
[Gratch  &  Marsella,  2004].  To  achieve  this  end,  we  base  our  work  on  the  broad  variables 
people  use  in  determining  social  causality  and  responsibility.  Psychological  and  philoso¬ 
phical  theories  largely  agree  on  these  basic  variables  though  they  differ  in  terminology.  In 
this  report,  we  adopt  the  terminology  of  Shaver  [1985].  In  Shaver’s  model,  the  judgment 
process  proceeds  by  assessing  several  key  variables:  who  caused  the  event;  Did  the  actor 
foresee  the  consequence;  Did  she  intend  to  bring  the  consequence  about;  Did  she  have 
choices  or  act  under  coercion  (e.g.,  by  an  authority)? 

Though  the  theory  identifies  the  conceptual  variables  for  social  causality  and  responsibil¬ 
ity  judgment,  in  modeling  social  behavior  of  intelligent  agents,  we  cannot  assume  that  an 
agent  has  privileged  access  to  the  mental  states  of  other  agents,  but  rather,  an  agent  can 


only  make  inferences  and  judgment  based  on  the  evidence  accessible  in  the  computa¬ 
tional  system  it  situates.  Current  intelligent  systems  are  increasingly  sophisticated,  usu¬ 
ally  involving  natural  language  conversation,  interactions  of  multiple  agents  and  a  plan¬ 
ning  module  to  plan  for  sequence  of  actions,  with  methods  that  explicitly  model  beliefs, 
desires  and  intentions  of  agents.  All  these  should  play  a  role  in  evaluating  the  conceptual 
variables  underlying  social  causality  and  responsibility  judgment. 

In  order  to  bridge  the  conceptual  descriptions  of  the  variables  and  the  computational  re¬ 
alization  in  application  systems,  we  need  to  model  the  inferential  mechanism  that  derives 
the  variable  values  needed  for  the  judgment  from  information  and  context  available  in 
practical  systems.  This  report  presents  a  domain-independent  computational  model  of 
social  causality  and  responsibility.  The  model  infers  the  key  variables  from  plan  knowl¬ 
edge  and  communication.  To  assess  the  veracity  of  the  approach  in  modeling  human  so¬ 
cial  inference,  we  conduct  empirical  studies  to  evaluate  and  compare  the  model  with  sev¬ 
eral  other  models  of  responsibility  and  blame. 

In  the  rest  of  the  report,  we  first  introduce  the  judgment  process  and  how  the  key  vari¬ 
ables  are  utilized  in  the  process,  and  then  present  the  computational  model.  We  finally 
evaluate  the  model  using  empirical  data  and  compare  our  approach  with  the  related  work. 

2.  Judgment  Process  and  Key  Variables 

We  base  our  work  on  the  most  influential  attributional  models  of  Shaver  [1985]  and 
Weiner  [1995]  for  social  causality  and  responsibility.  Their  models  suggest  that  physical 
causality  and  coercion  identify  who  is  responsible  for  some  outcome  under  evaluation, 
whereas  mental  factors,  intention  and  foreseeability,  determine  how  much  responsibility 
and  blame/credit  are  assigned. 

The  evaluations  of  physical  causality  and  coercion  identify  the  responsibility  party. 
Physical  causality  refers  to  the  connection  between  actions  and  the  effects  they  produce. 
In  the  absence  of  external  coercion,  the  actor  whose  action  directly  produces  the  outcome 
is  regarded  as  responsible.  However,  in  social  situations,  an  agent  may  cause  an  outcome 
because  she  could  not  have  done  otherwise.  Coercion  occurs  when  some  external  force, 
such  as  a  more  powerful  individual  or  a  socially  sanctioned  authority,  limits  an  agent’s 
freedom  of  choice.  The  presence  of  coercion  can  deflect  some  or  all  of  the  responsibility 
to  the  coercive  force,  depending  on  the  perceived  degree  of  coercion. 

Intention  and  forseeability  determine  the  degree  of  responsibility  and  blame.  Intention  is 
generally  conceived  as  the  commitment  to  work  towards  a  certain  act  or  outcome.  Most 
theories  view  intention  as  the  major  determinant  of  the  degree  of  responsibility.  If  an 
agent  intends  an  action  to  achieve  an  outcome,  then  the  agent  must  have  the  foreknowl¬ 
edge  that  the  action  brings  about  the  outcome.  Foreseeability  refers  to  an  agent’s  fore¬ 
knowledge  about  actions  and  their  consequences.  The  higher  the  degree  of  intention,  the 
greater  the  responsibility  assigned.  The  lower  the  degree  of  foreseeability,  the  less  the 
responsibility  assigned. 


An  agent  may  intentionally  perform  an  aetion,  but  may  not  intend  all  the  aetion  effects.  It 
is  outcome  intent  (i.e.,  intentional  action  effect),  rather  than  act  intentionality  (i.e.,  inten¬ 
tional  action)  that  are  key  in  responsibility  judgment  [Weiner,  2001].  Similar  difference 
exists  in  outcome  coercion  (i.e.,  coerced  action  effect)  and  act  coercion  (i.e.,  coerced 
action).  An  agent’s  intentional  action  and  action  effect  may  succeed  or  fail.  However,  as 
long  as  it  manifests  intentions,  a  failed  attempt  can  be  blamed  or  credited  almost  the  same 
as  a  successful  one  [Zimmerman,  1988]. 

The  result  of  the  judgment  process  is  the  assignment  of  certain  blame  or  credit  to  the  re¬ 
sponsible  agent(s).  The  intensity  of  blame  or  credit  is  determined  by  the  degree  of  re¬ 
sponsibility  as  well  as  the  severity  or  positivity  of  the  outcome.  The  degree  of  responsi¬ 
bility  is  based  on  the  assessed  values  of  attribution  variables. 

3.  The  Social  Inference  Model 

We  build  a  computational  model  of  this  social  judgment  process,  showing  how  auto¬ 
mated  methods  for  causal  and  dialogue  reasoning  can  provide  a  mechanistic  explanation 
of  how  people  arrive  at  judgments  of  blame  and  responsibility.  Here  we  briefly  summa¬ 
rize  the  model.  The  reader  may  refer  to  [Mao  &  Gratch,  2003a,  2003b,  2004a,  2004b]  for 
details. 


3.1  Modular  Structure 

The  judgment  of  social  causality  and  responsibility  is  a  subjective  process.  It  is  from  the 
perspective  of  a  perceiving  agent  (i.e.,  the  agent  who  makes  the  judgment),  and  based  on 
the  perceiver’s  interpretation  of  the  significance  of  events.  The  perceiver  uses  her  own 
knowledge  about  the  observed  agents’  behavior  to  infer  certain  beliefs  (in  terms  of  the 
key  variables).  The  inferred  variable  values  are  then  applied  to  the  judgment  process  to 
form  an  overall  result. 

Two  important  sources  of  information  contribute  to  the  inference  of  key  variables.  One 
source  is  general  beliefs  about  actions  and  their  effects.  The  other  is  observations  of  the 
actions  performed  by  the  observed  agents,  including  physical  and  communicative  acts 
(e.g.,  in  a  conversational  dialogue).  The  inference  process  acquires  beliefs  from  commu¬ 
nicative  events  (i.e.,  dialogue  inference)  and  from  the  causal  information  about  the  ob¬ 
served  action  execution  (i.e.,  causal  inference).  To  construct  a  computational  model,  we 
need  to  represent  such  information  and  make  inferences  over  it.  We  also  need  an  algo¬ 
rithm  to  describe  the  overall  judgment  process. 

We  have  designed  a  modular  structure  for  evaluating  social  causality  and  responsibility 
(i.e.,  a  social  inference  module),  and  its  interface  with  other  system  components.  Figure  1 
illustrates  the  structure  of  the  module.  It  takes  the  observed  communicative  events  and 
executed  actions  as  inputs.  Causal  information  and  social  information  are  also  important 
inputs.  Causal  information  includes  an  action  theory  and  a  plan  library  (discussed  below). 
Social  information  specifies  social  roles  and  the  power  relationship  of  the  roles.  The  in¬ 
ference  process  first  applies  dialogue  inference,  and  then  causal  inference.  Both  make  use 


of  the  commonsense  heuristics,  and  derive  beliefs  about  the  variable  values.  The  values 
are  then  served  as  inputs  of  the  algorithm,  which  determines  responsibility,  and  assigns 
certain  blame  or  credit  to  the  responsible  agents. 


Figure  1  Stmcture  of  the  Social  Inference  Module 

3,2  Computational  Representation 

To  represent  an  agent’s  causal  knowledge,  we  have  adopted  a  hierarchical  plan  represen¬ 
tation  used  in  many  intelligent  agent  systems.  This  representation  provides  a  concise  de¬ 
scription  of  the  physical  causal  relationship  between  events  and  world  states.  It  also  pro¬ 
vides  a  clear  structure  for  exploring  alternative  courses  of  actions  and  detecting  plan  in¬ 
teractions. 

Actions  and  Plans 

Physical  causality  is  encoded  via  a  hierarchical  plan  representation.  Actions  consist  of  a 
set  of  propositional  preconditions  and  effects  (including  conditional  effects).  Each  action 
step  is  either  a  primitive  action  (i.e.,  an  action  directly  executable  by  some  agent)  or  an 
abstract  action.  An  abstract  action  may  be  decomposed  in  alternative  ways  and  the  ef¬ 
fects  of  an  abstract  action  depend  on  these  alternatives.  For  example,  if  there  are  two  al¬ 
ternatives  for  performing  an  abstract  action,  only  those  effects  that  occur  in  each  alterna¬ 
tive  are  necessarily  the  effects  of  the  abstract  act.  The  desirability  of  action  effects  is  rep¬ 
resented  by  utility  values  [Blythe,  1999]. 

A  plan  is  a  set  of  actions  to  achieve  certain  intended  goal(s).  As  a  plan  may  contain  ab¬ 
stract  actions  (i.e.,  an  abstract  plan),  each  abstract  plan  indicates  a  plan  structure  of  de¬ 
composition.  Decomposing  the  abstract  actions  into  primitive  ones  in  an  abstract  plan 
results  in  a  set  of  primitive  plans  (i.e.,  plans  with  only  primitive  actions),  which  is  di¬ 
rectly  executable  by  agents.  In  addition,  each  action  in  the  plan  structure  is  associated 
with  the  performer  (i.e.,  agents  capable  of  performing  the  action)  and  the  authority  (i.e., 
agent  who  authorizes  the  action  execution).  The  performer  cannot  execute  the  action  until 


authorization  is  given  by  the  authority.  This  represents  the  hierarehieal  organizational 
strueture  of  soeial  agents. 


Communicative  Events 

Natural  language  eommunieation  is  a  rieh  information  souree  for  inferring  attribution 
variables.  We  assume  eonversations  between  agents  are  grounded  and  they  eonform  to 
Griee’s  maxims  of  Quality  and  Relevance  (i.e.,  true  and  relevant  information  exehange  in 
eonversation).  We  represent  eommunieative  events  as  speech  act  [Austin,  1962]  se- 
quenee,  and  analyze  the  following  aets  that  are  typieal  in  negotiation  dialogues  [Traum  et 
al,  2003],  inform,  order,  request,  accept,  reject,  and  counter-propose. 

3,3  Inferences 

The  inferenee  of  physieal  causality,  coercion,  intentions  and  foreknowledge  is  informed 
by  dialogue  and  causal  evidence  in  social  interactions.  We  introduce  commonsense  heu¬ 
ristics  that  allow  an  agent  to  make  inferences  based  on  this  evidence. 

Agency 

A  first  step  in  attributing  responsibility  and  blame  is  to  identify  which  actors’  actions 
contribute  to  the  occurrence  of  an  outcome  under  evaluation.  In  a  multi-agent  plan  execu¬ 
tion  environment,  an  actor  can  produce  an  outcome  through  the  assistance  of  other 
agents.  These  other  agents  are  viewed  as  indirect  agency.  Given  a  specific  outcome  p  and 
the  observed  action  set  S,  the  following  actions  in  S  are  relevant  to  achieving  p: 

•  The  primitive  action  A  that  has  p  as  its  effect. 

•  The  actions  that  establish  a  precondition  of  a  relevant  action  to  achieving  p. 

•  If  or  a  precondition  of  a  relevant  action  is  enabled  by  the  consequent  of  a  condi¬ 
tional  effect,  the  actions  that  establish  the  antecedent  of  the  conditional  effect  are 
relevant. 

These  relevant  actions  are  the  possible  causes  of  the  outcome  p.  Therefore,  their  perform¬ 
ers  are  potentially  responsible  for  p. 

Coercion 

An  actor  could  be  absolved  of  responsibility  if  she  was  coerced  by  other  forces,  but  just 
because  an  agent  applies  coercive  force  does  not  mean  coercion  actually  occurs.  What 
matters  is  whether  this  force  truly  constrains  the  actor’s  freedom  to  avoid  the  outcome. 
Causal  inference  helps  evaluate  outcome  coercion  from  evidence  of  act  coercion. 

Two  concepts  are  important  in  understanding  coercion.  One  concept  is  social  obligation, 
created  by  utterance,  role  assigned,  etc.  The  other  is  {un)willingness.  For  example,  if 
some  authorizing  agent  commands  another  agent  to  perform  a  certain  action,  then  the 
latter  agent  has  an  obligation  to  do  so.  But  if  the  agent  is  actually  willing  to,  this  is  a  vol¬ 
untary  act  rather  than  a  coercive  one. 


If  there  is  no  elear  evidenee  that  an  agent  intends  beforehand,  and  the  agent  aeeepts  her 
obligation,  there  is  evidenee  of  eoereion.  In  this  inferenee  rule,  intend(x,  p,  tl)  represents 
that  agent  x  intends  that  proposition p  at  time  tl,  obligation(x,  p,  y,  t2)  represents  that  x  has 
an  obligation  p  by  agent  y  at  time  t2,  aeeept(x,  p,  t3)  represents  that  x  aeeepts  that  p  at 
time  t3,  and  eoeree(y,  x,p,  t4)  represents  thaty  eoerees  x  that  p  at  time  t4. 

-^(3tl){tl<t3  A  intend(x,  p,  tl))  a  obligation(x,  p,  y,  t2)  a  aeeept(x,  p,  t3)  a  t2<t3<t4  ^ 

eoeree(y,  x,  p,  t4) 

In  another  ease,  when  there  is  elear  evidenee  of  the  unwillingness  (i.e.,  intend(x, />,  tl)  is 
false),  there  is  strong  evidenee  of  eoereion. 

Given  the  aetion  preeonditions  are  initially  true,  if  an  agent  is  eoereed  to  exeeute  a  primi¬ 
tive  aetion,  the  agent  is  also  eoereed  to  aehieve  all  the  aetion  effeets.  If  being  eoereed  to 
exeeute  an  abstraet  aetion  and  the  aetion  has  only  one  deeomposition,  then  the  agent  is 
also  eoereed  to  exeeute  the  sub-aetions  and  aehieve  all  the  sub-aetion  effeets.  If  the  eo¬ 
ereed  aetion  has  multiple  deeompositions,  then  the  agent  has  options:  only  the  effeets 
appear  in  all  alternatives  are  unavoidable,  and  thus  these  effeets  are  eoereed;  Sinee  other 
effeets  that  only  appear  in  some  (but  not  all)  alternatives  are  avoidable,  they  are  not  eo¬ 
ereed. 

If  some  agents  bloek  other  aetion  alternatives  (by  disabling  aetion  preeonditions),  the 
only  alternative  left  as  well  as  its  effeets  are  eoereed.  These  bloeking  agents  are  also 
viewed  as  eoereers.  If  a  eonditional  effeet  is  eoereed  and  its  anteeedent  is  initially  true  or 
enabled  by  other  agents,  then  its  eonsequent  is  eoereed.  These  other  agents  are  also 
viewed  as  eoereers.  Otherwise,  the  eonsequent  is  not  eoereed. 

Intentions 

Intentions  play  a  eentral  role  in  determining  the  degree  of  responsibility  and  blame  as¬ 
signments.  Aet  and  outeome  intentions  ean  be  inferred  from  eonversation  eommunieation 
between  agents.  For  example,  an  order  or  a  request  shows  the  speaker’s  intent.  The  two 
speeeh  aets  have  different  implieations  on  the  soeial  status  between  the  speaker  and  the 
hearer.  If  an  order  is  sueeessfully  issued  to  a  subordinate,  it  ereates  a  soeial  obligation  for 
the  subordinate  to  perform  the  eontent  of  the  aet.  The  hearer  may  accept,  reject  or 
counter-propose.  Various  inferenees  ean  be  made  depending  on  the  response  of  the 
hearer  and  the  power  relationship  between  the  speaker  and  the  hearer.  For  example,  if  the 
hearer  eounters  the  order,  and  proposes  another  alternative,  it  ean  be  inferred  that  both  the 
speaker  and  the  hearer  know  the  alternatives.  It  is  also  believed  that  the  hearer  does  not 
intend  what  is  ordered,  but  want  the  alternative.  If  the  speaker  has  known  the  alternatives 
yet  still  orders  one,  infer  that  the  speaker  intends  the  ehosen  aetion  but  not  the  alternative. 
The  reader  may  refer  to  [Mao  &  Grateh,  2003b]  for  the  eomplete  rules. 

intend(5,  p,  tl)  a  — iobligation(/z,  p,  s,  t2)  a  aeeept(/z,  p,  t3)  a  tl<t3  a  t2<t3<t4  ^  in- 

tend(/2,  p,  t4) 


Outcome  intent  ean  also  be  partially  inferred  from  evidenee  of  aet  intentionality.  For  ex¬ 
ample,  if  an  agent  intends  an  aetion  voluntarily,  the  agent  must  intend  at  least  one  aetion 
effeet.  If  there  is  only  one  aetion  effeet  (signifieant  to  the  agent),  we  ean  exaetly  infer 
whieh  effeet  the  agent  intends.  As  plans  provide  eontext  in  evaluating  intention,  with 
assoeiation  to  the  goals  and  reasons  of  an  agent’s  behavior,  in  the  absenee  of  elear  evi- 
denee  from  dialogue  inferenee,  we  employ  a  general  plan-based  algorithm  to  reeognize 
intentions  [Mao  &  Grateh,  2004b]. 

Foreknowledge 

Sinee  foreknowledge  refers  to  an  agent’s  epistemie  state,  it  is  mainly  derived  from  dia¬ 
logue  inferenee.  For  example,  inform  gives  evidenee  that  the  eonversants  know  the  eon- 
tent  of  the  aet.  Besides,  intention  reeognition  also  helps  infer  an  agent’s  foreknowledge, 
as  intentions  entail  foreknowledge  {Axiom  4  in  [Mao  &  Grateh,  2004a]). 

3,4  Algorithm 

The  judgment  proeess  begins  with  some  speeifie  outeome  that  is  under  evaluation,  and 
the  judgment  result  is  based  on  the  inferenees  of  variable  values  introdueed  above.  The 
aequired  values  for  ageney  and  eoereion  eontribute  to  the  evaluation  of  responsible 
agents.  We  have  developed  an  algorithm  for  traeing  the  responsible  agents  [Mao  & 
Grateh,  2003b].  The  algorithm  starts  with  the  primitive  aetion  that  direetly  eauses  the 
evaluated  outeome  and  works  up  the  plan  hierarehy.  During  eaeh  loop,  it  applies  infer¬ 
enee  rules  and  intention  reeognition  method  to  reason  about  attribution  variables.  If  out¬ 
eome  eoereion  is  true,  the  algorithm  proeeeds  until  reaehing  the  root  of  the  plan  hierar¬ 
ehy.  In  the  meantime,  the  applieation  of  inferenee  rules  and  intention  reeognition  algo¬ 
rithm  aequires  beliefs  for  foreknowledge  and  aet/outeome  intentions,  whieh  determine 
the  intensity  of  responsibility  and  blame/eredit. 

4.  Evaluation  and  Comparison 

Our  claim  of  evaluation  is  that  this  model  will  better  prediet  human  judgments  of  respon¬ 
sibility  and  blame  than  other  potential  approaehes.  Here,  we  report  the  results  of  an  ex¬ 
periment  eomparing  our  model  and  three  eomputational  alternatives  to  human  data. 

4,1  Alternative  Models 

It  is  not  uneommon  to  use  physieal  eausality  as  a  substitute  for  modeling  soeial  eausality. 
This  was  the  approaeh  used,  for  example,  in  the  MRE  team  training  system  [Riekel  et  al, 
2002].  A  simple  cause  model  always  assigns  responsibility  and  blame  to  the  aetor  whose 
aetion  direetly  produees  the  outeome. 

Instead  of  always  pieking  up  the  aetor,  a  slightly  more  sophistieated  model  ean  ehoose 
the  highest  authority  (if  there  is  one)  as  the  responsible  and  blameworthy  agent.  We  eall 
sueh  model  simple  authority  model. 


Chockler  and  Halpern  [2004]  propose  a  structural-model  approach  to  responsibility  and 
blame  (abbreviated  to  C&H  model  below).  They  give  a  definition  of  responsibility,  which 
extends  the  definition  of  causality  introduced  by  Halpern  and  Pearl  [2001].  For  example, 
if  a  person  wins  an  election  11-0,  then  each  voter  who  votes  for  her  is  a  cause  for  the  vic¬ 
tory,  but  each  voter  is  less  responsible  for  the  victory  than  if  she  had  won  6-5.  Based  on 
this  notion  of  responsibility,  they  then  defined  the  degree  of  blame,  using  the  expected 
degree  of  responsibility  weighed  by  the  epistemic  state  of  an  agent. 

4.2  Method 

Our  model  argues  that  people  will  view  blame  differently  based  on  their  perception  of 
key  variables  such  as  intentions  and  coercion.  Thus  a  good  test  is  to  see  how  the  models 
perform  when  such  variables  are  systematically  manipulated.  We  compare  attributions  of 
blame  by  the  four  models  with  human  judgments  using  four  variants  of  the  “firing  squad” 
scenario  in  [Chockler  &  Halpern,  2004]. 

Scenario  1  is  the  original  example;  There  is  a  ten-man  firing  squad.  Only  one  marksman 
has  live  bullets  in  his  rifle;  the  rest  have  blanks.  The  marksmen  do  not  know  who  has  the 
live  bullets.  They  shoot  at  the  prisoner  and  the  death  occurs.  Scenario  2  extends  the  ex¬ 
ample  to  include  an  authority  -  the  commander,  who  orders  the  squad  to  shoot.  Scenario 
3  further  extends  the  example  by  presenting  a  negotiation  dialogue  between  the  com¬ 
mander  and  the  marksmen.  The  marksmen  first  reject  the  commander’s  order.  The  com¬ 
mander  insists  and  orders  again.  Finally  the  squad  accepts  the  order  and  shoot.  In  Sce¬ 
nario  4,  the  commander  still  orders.  However,  each  marksman  has  freedom  to  choose 
either  using  blanks  or  live  bullets  before  shooting. 

In  each  scenario,  we  query  27  subjects  (mostly  university  staffs  including  graduates,  with 
ages  ranging  from  20  to  45  and  evenly  distributed  genders)  to  assess  their  judgments  of 
responsibility,  blame  and  coercion. 

4.3  Results 

Figure  2  shows  proportions  of  the  subjects  that  attribute  blame  and  responsibility  to  dif¬ 
ferent  categories  of  agents  in  the  scenarios,  and  corresponding  confidence  intervals  for 
large  population  (a=0.05)  [Rice,  1994].  For  example,  in  scenario  1,  3  subjects  blame  the 
marksman  with  live  bullets  in  his  rifle,  19  blames  all  the  marksmen  and  the  rest  do  not 
blame  any  of  them.  The  analysis  of  the  sample  data  and  their  confidence  intervals  show 
that  a  small  percentage  of  the  population  will  blame  the  marksman  with  live  bullets,  a 
significant  majority  will  blame  all  the  marksmen,  and  a  small  percentage  won’t  blame 
any,  with  0.95  confidence. 
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Figure  2  Proportions  of  Population  Agreement  on  Responsibility/Blame  in  Sample  Scenarios 
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Table  1  Comparison  of  Results  by  Different  Models  with  Human  Data 


Table  1  shows  the  results  on  blame  generated  by  different  models.  All  the  results  are 
compared  with  the  dominant  proportions  (i.e.  majority)  of  people’s  agreement  (though  in 
Scenario  4,  there  is  an  overlap  between  two  categories.  That’s  why  we  note  our  model  as 


a  partial  match).  The  simple  eause  model  always  ehooses  physieal  eausality.  It  only  par¬ 
tially  matehes  the  human  agreement  in  Scenario  4,  but  is  ineonsistent  with  the  data  in 
Scenarios  1-3.  Simple  authority  model  always  pieks  up  the  highest  authority.  It  matehes 
the  human  data  in  Scenario  2  and  3,  but  is  ineonsistent  with  the  data  in  other  scenarios.  In 
general,  simple  models  are  insensitive  to  the  ehanging  situation  speeified  in  eaeh  see- 
nario. 

C&H  model  matches  human  judgments  only  in  Scenario  1.  In  the  remaining  scenarios,  its 
results  are  ineompatible  with  the  data.  Like  other  work  in  eausality  researeh,  the  underly¬ 
ing  eausal  reasoning  in  C&H  model  is  based  on  philosophical  prineiples  (i.e.,  eounterfae- 
tual  dependeneies).  Their  extended  definition  of  responsibility  aeeounts  better  for  multi¬ 
ple  eauses  and  the  extent  to  whieh  each  cause  eontributes  to  the  oecurrenee  of  an  out- 
eome.  However,  the  results  show  that  their  blame  model  does  not  match  human  data  well. 
These  empirieal  findings  generally  support  our  hypothesis. 

In  the  next  section,  we  discuss  how  our  model  appraises  each  scenario  and  compare  our 
approaeh  with  C&H  model. 

4.4  Comparison  and  Discussions 

Scenario  1 

Actions  and  plans  are  explicitly  represented  in  our  approach.  In  Scenario  1,  each  marks¬ 
man  performs  a  primitive  action,  shooting.  The  action  has  a  conditional  effect,  with  the 
antecedent  live  bullets  and  the  consequent  death.  All  marksmen’s  shooting  actions  consti¬ 
tute  a  team  plan  squad  firing,  with  outcome  death.  The  team  plan  is  observed  executed, 
and  plan  outcome  occurs.  Applying  our  intention  recognition  algorithm^  [Mao  &  Gratch, 
2004b],  the  marksmen  are  believed  to  intend  the  actions  and  the  only  outcome.  The 
marksman  with  the  bullets  is  the  sole  cause  of  the  death.  This  marksman  intends  the  out¬ 
come,  and  thus  deserves  high  degree  of  responsibility  and  blame.  As  other  marksmen 
with  blanks  also  intend  the  actions  and  the  outcome,  and  shooting  actions  are  observed 
executed  but  the  antecedent  of  the  conditional  effect  is  false,  their  failed  attempt  can  be 
detected.  Therefore,  other  marksmen  are  also  blameworthy  for  their  attempt  (recall  that 
an  agent  can  be  blamed/credited  for  a  successfully  produced  outcome  as  well  as  for  an 
unsuccessful  attempt). 

C&H  model  judges  responsibility  according  to  the  actual  cause  of  the  event.  As  the 
marksman  with  the  bullets  is  the  only  cause  of  the  death,  this  marksman  has  degree  of 
responsibility  1  for  the  death  and  others  have  degree  of  responsibility  0.  This  result  is 
inconsistent  with  human  data.  In  determining  blame,  C&H  model  draws  the  same  conclu¬ 
sion  as  ours,  but  their  approach  is  different.  They  consider  each  marksman’s  epistemic 
state  before  action  performance  (corresponding  to  foreknowledge).  There  are  10  situa¬ 
tions  possible,  depending  on  who  has  the  bullets.  Each  marksman  is  responsible  for  one 


*  Note  that  our  intention  recognition  algorithm  is  generally  applied  to  a  plan  library  with  multiple  plans  and  sequences 
of  actions,  which  is  typical  in  intelligent  agent  applications.  In  this  oversimplified  example,  intention  recognition  be¬ 
comes  trivial. 


situation  with  degree  of  responsibility  1.  Given  that  each  situation  is  equally  likely  to 
happen  (1/10  possibility),  each  marksman  has  degree  of  blame  1/10. 

As  there  is  no  notion  of  intention  in  their  model,  C&H  model  uses  foreknowledge  as  the 
only  determinant  for  blame  assignment.  This  is  fine  when  there  is  no  foreknowledge,  as 
no  foreknowledge  entails  no  intention.  However,  when  there  is  foreknowledge,  the  blame 
assigned  is  high,  even  if  there  might  be  no  intentions  in  the  case.  For  example,  if  a 
marksman  fires  the  gun  by  mistake,  without  any  intention  of  shooting  or  attempting  the 
death,  in  C&H  model,  still  he  will  be  blamed  just  the  same  as  those  who  intend. 

Scenarios  2  &  3 

In  our  model,  we  take  different  forms  of  social  interactions  into  account.  The  inference 
process  reasons  about  the  beliefs  from  both  causal  and  dialogue  evidence.  Figure  3  illus¬ 
trates  the  team  plan  of  the  squad  in  Scenarios  2  and  5,  where  a  commander  acts  as  an 
authority  of  the  squad  {AND  denotes  that  the  action  has  only  one  decomposition). 


Figure  3  Team  Plan  of  the  Squad  in  Scenarios  2  and  3 


The  intermediate  inference  results  for  Scenario  2  are  given  below  {cmd,  sqd  and  mkn 
stand  for  the  commander,  the  squad  and  the  marksman,  respectively.  Beliefs  are  ordered 
by  time). 


(1)  intend(cmJ,  do{sqd,  firing)) 

(2)  oh\igation{sqd,firing,  cmd) 

(3)  mtQnd{cmd,  death) 

(4)  coerce(cmJ,  sqd,  firing) 

(5)  coerce(cmJ,  mkn,  shooting) 

(6)  coerce(cmt/,  mkn,  death) 


(Act  order) 

(Act  order) 

(Rule  for  intention  &  Result  1) 
(Act  accept  &  Result  2) 

(Rule  for  coercion) 

(Rule  for  coercion) 


So  in  Scenario  2,  the  marksmen  cause/attempt  the  death  due  to  coercion.  The  commander 
is  responsible  for  the  death.  As  the  commander  intends  the  outcome,  the  commander  is  to 
blame  with  high  intensity. 

Scenario  3  includes  a  sequence  of  negotiation  acts.  The  above  beliefs  4-6  thus  change  to 
the  following; 


(4)  -^miQnd{sqd,firing)  (Act  reject  and  Result  1) 

(5)  coerce(cmJ,  sqd,  firing)  (Act  accept  and  Results  2  &  4) 

(6)  coerce(cmJ,  mkn,  shooting)  (Rule  for  coercion) 

(!)  coeree(cm<i,  mkn,  death)  (Rule  for  coercion) 

Clearly  the  marksmen  do  not  intend  firing.  Scenario  3  shows  strong  coercion.  This  is  also 
reflected  in  the  data.  More  proportions  of  people  regard  the  commander  as  responsible 
and  blameworthy  in  Scenario  3  than  in  Scenario  2. 

C&H  model  represents  all  the  relevant  events  in  the  seenarios  as  random  variables.  So  if 
we  want  to  model  the  communieative  acts  in  Scenarios  2  and  3,  each  act  would  be  a  sepa¬ 
rate  variable  in  their  model.  This  is  problematic  when  conversational  dialogue  is  involved 
in  a  scenario.  As  the  approach  uses  the  structural  equations  to  represent  the  relationships 
between  variables,  and  each  equation  in  the  model  must  be  deterministic,  it  is  difficult  to 
come  up  with  such  equations  for  a  dialogue  sequenee.  For  example,  if  we  want  to  model 
communicative  acts  in  Scenario  3,  we  will  have  to  give  deterministic  relationship  be¬ 
tween  them  (e.g.,  if  the  eommander  orders,  the  squad  will  aecept).  Sueh  strict  equations 
simply  do  not  exist  in  a  natural  conversation.  If  we  ignore  some  communicative  aets  in 
between,  important  information  conveyed  by  these  aets  will  be  lost. 

Assume  marksman- 1  is  the  one  with  the  live  bullets.  Using  C&H  approaeh,  the  outcome 
is  counterfactually  depends  on  marksman-1 ’s  shooting,  so  marksman-1 ’s  shooting  is  an 
actual  cause  of  the  death.  Similarly,  the  commander’s  order  is  also  an  aetual  eause  of  the 
death.  Based  on  the  responsibility  definition  in  C&H  model,  both  the  commander  and 
marksman- 1  are  responsible  for  the  death,  and  eaeh  has  degree  of  responsibility  1.  This 
result  is  ineonsistent  with  human  data.  In  assigning  blame,  there  are  ten  situations  alto¬ 
gether,  and  in  each  situation,  the  commander  has  expected  responsibility  1,  so  the  com¬ 
mander  is  to  blame  with  degree  1.  The  marksmen  each  has  degree  of  blame  1/10.  Thus 
C&H  model  appraises  that  the  eommander  and  all  marksmen  are  blameworthy  for  the 
outcome.  As  their  model  for  responsibility  and  blame  is  the  extension  of  counterfactual 
eausal  reasoning,  which  has  been  eritieized  as  far  too  permissive  [Hopkins  &  Pearl, 
2003],  the  same  problem  is  also  refieeted  in  their  model  of  responsibility  and  blame. 

Scenario  4 

Different  from  the  previous  seenarios,  in  Scenario  4,  the  bullets  are  not  initially  set  before 
the  scenario  starts.  The  marksmen  can  choose  to  use  either  bullets  or  blanks  before  shoot¬ 
ing.  Firing  is  still  the  joint  aetion  of  the  squad,  but  there  is  no  team  plan  or  common  goal 
for  the  squad.  As  the  eommander  orders  the  joint  aetion,  aet  eoercion  is  true.  However, 
based  on  the  rules  of  inferring  outeome  eoereion  from  act  coercion,  the  marksmen  are  not 
eoerced  the  outeome.  So  in  this  case,  the  commander  is  not  responsible  for  the  outcome, 
but  rather,  the  marksmen  who  choose  to  use  bullets  and  eause  the  death  are  responsible 
and  blameworthy.  Figure  2  shows  that  in  Scenario  4,  people’s  judgments  somehow  dif¬ 
fuse.  There  is  an  overlap  between  blaming  the  marksmen  with  bullets  and  blaming  both 
the  eommander  and  the  marksmen  with  bullets.  Nonetheless,  the  category  our  model  falls 
into  is  elearly  better  than  the  rest  three. 


C&H  model  requires  all  the  struetural  equations  to  be  deterministie.  In  essenee,  their 
model  eould  not  handle  alternative  eourses  of  aetions,  whieh  inherently  have  nondeter- 
ministic  property.  One  way  to  eompensate  for  this  is  to  push  the  nondeterminism  into  the 
setting  of  the  eontext.  For  example,  in  Scenario  4,  they  eould  build  a  eausal  model  to  let 
the  eontext  determine  whether  the  bullets  are  live  or  blank  for  eaeh  marksman,  and  then 
have  a  probability  distribution  over  eontexts.  After  that,  they  ean  eompute  the  probability 
of  an  aetual  eause.  However,  sinee  these  eontexts  are  only  baekground  variables,  their 
probabilities  eould  not  aetually  impaet  the  reasoning  proeess  per  se. 

5.  Summary 

Intelligent  virtual  agents  are  typieally  embedded  in  a  soeial  environment  and  must  reason 
about  soeial  eause  and  effeet.  Soeial  eausal  reasoning  is  qualitatively  different  from 
physieal  eausal  reasoning  that  underlies  most  eurrent  intelligent  systems.  In  this  report, 
we  review  a  general  eomputational  model  of  soeial  eausality  and  responsibility.  Our  ap- 
proaeh  bases  on  the  broad  features  people  use  in  behavior  judgment,  ineluding  physieal 
eause,  intentions,  foreknowledge  and  eoereion.  We  present  how  our  model  reasons  about 
beliefs  about  attribution  variables  for  the  judgment  proeess,  and  empirieally  evaluate  and 
eompare  the  model  with  several  other  approaehes. 

The  initial  results  show  that  our  model  better  prediets  human  judgment  of  blame  and  re¬ 
sponsibility  than  other  potential  approaehes.  Our  future  work  needs  to  further  refine  the 
model  and  eonduet  more  experiments  to  systematieally  evaluate  the  eapabilities  of  the 
model. 

Acknowledgements 

The  projeet  or  effort  deseribed  here  has  been  sponsored  by  the  U.S.  Army  Researeh,  De¬ 
velopment,  and  Engineering  Command  (RDECOM).  We  would  like  to  thank  Joseph 
Halpern  and  Andrew  Gordon  for  the  helpful  diseussions.  Statements  and  opinions  ex¬ 
pressed  do  not  neeessarily  refleet  the  position  or  the  poliey  of  the  United  States  Govern¬ 
ment,  and  no  offieial  endorsement  should  be  inferred. 

References 

J.  Austin.  How  to  Do  Things  with  Words.  Harvard  University  Press,  1962. 

J.  Blythe.  Deeision-Theoretie  Planning.  AI Magazine,  20(2);37-54,  1999. 

H.  Choekler  and  J.  Y.  Halpern.  Responsibility  and  Blame:  A  Struetural-Model  Approaeh. 
Journal  of  Artificial  Intelligence  Research,  22:93-1 15,  2004. 

J.  Grateh  and  S.  Marsella.  A  Domain-Independent  Eramework  for  Modeling  Emotion. 
Journal  of  Cognitive  Systems  Research,  5(4):269-306,  2004. 


J.  Y.  Halpern  and  J.  Pearl.  Causes  and  Explanations;  A  Struetural-Model  Approaeh  -  Part 
I;  Causes.  Proceedings  of  the  Seventeenth  Conference  in  Uncertainty  in  Artificial  Intelli¬ 
gence,  2001. 

M.  Hopkins  and  J.  Pearl.  Clarifying  the  Usage  of  Struetural  Models  for  Commonsense 
Causal  Reasoning.  Proceedings  of  AAAI  Spring  Symposium  on  Logic  Formulizations  of 
Commonsense  Reasoning,  2003. 

W.  Mao  and  J.  Grateh.  The  Soeial  Credit  Assignment  Problem.  Proceedings  of  the 
Fourth  International  Working  Conference  on  Intelligent  Virtual  Agents,  2003a. 

W.  Mao  and  J.  Grateh.  The  Social  Credit  Assignment  Problem  (Extended  Version).  ICT 
Technical  Report  ICT-TR-02-2003,  2003b. 

W.  Mao  and  J.  Grateh.  Social  Judgment  in  Multiagent  Interactions.  Proceedings  of  the 
Third  International  Joint  Conference  on  Autonomous  Agents  and  Multiagent  Systems, 
2004a. 

W.  Mao  and  J.  Grateh.  A  Utility-Based  Approach  to  Intention  Recognition.  AAMAS  2004 
Workshop  on  Agent  Tracking:  Modeling  Other  Agents  from  Observations,  2004b. 

J.  A.  Rice.  Mathematical  Statistics  and  Data  Analysis  (Second  Edition).  Duxbury  Press, 
1994. 

J.  Rickel,  S.  Marsella,  J.  Grateh,  R.  Hill,  D.  Traum  and  W.  Swartout.  Toward  a  New 
Generation  of  Virtual  Humans  for  Interactive  Experiences.  IEEE  Intelligent  Systems, 
I7(4):32-38,  2002. 

K.  G.  Shaver.  The  Attribution  of  Blame:  Causality,  Responsibility  and  Blameworthiness . 
Springer-Verlag,  1985. 

D.  Traum,  J.  Rickel,  J.  Grateh  and  S.  Marsella.  Negotiation  over  Tasks  in  Hybrid  Human- 
Agent  Teams  for  Simulation-Based  Training.  Proceedings  of  the  Second  International 
Joint  Conference  on  Autonomous  Agents  and  Multiagent  Systems,  2003. 

B.  Weiner.  Judgments  of  Responsibility:  A  Foundation  for  a  Theory  of  Social  Conduct. 
The  Guilford  Press,  1995. 

B.  Weiner.  Responsibility  for  Social  Transgressions:  An  Attributional  Analysis.  In;  B.  E. 
Malle,  E.  J.  Moses  and  D.  A.  Baldwin  (Ed.).  Intentions  and  Intentionality:  Foundations 
of  Social  Cognition,  pp.  331-344.  The  MIT  Press,  2001. 

M.  J.  Zimmerman.  An  Essay  on  Moral  Responsibility.  Rowman  &  Eittlefield,  1988. 


